-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Isolate ICP and FCP graphs using the runahead limit #5036
Conversation
96ba7a7
to
3823517
Compare
Assigning you as first reviewer @oliver-sanders, since it is was your idea to get "start-up graphs" cheaply and safely by manipulating the runahead limit (and a fine idea it was). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good feature, I might implore our operations team to switch to this.
Would be good if there was a short hand way of specifying the ICP without knowing it via the CLI (i.e. if we had to reflow the prep tasks).
Tested locally before and after (True/False), and found working 👍
@@ -34,10 +34,13 @@ ones in. --> | |||
|
|||
### Enhancements | |||
|
|||
[#5036](https://github.com/cylc/cylc-flow/pull/5036) - optional isolation of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Capitalise "optional" (and 5032 below it)?
Not sure I like this. One of the key features of Cylc is the support for parallel cycles. In this example, if One way around this is to make the start and end tasks run on separate cycles:
(syntax provided by @oliver-sanders !) It would be much nicer if we could do something like:
I guess that's a lot more work? |
Haha, that was my original suggestion, albeit explicitly restricted to start and finish graphs - you might have chimed in back on #4903!!
Well, I can think about it, if you and @oliver-sanders agree - he didn't like the separate graphs approach, which is how we ended up here. But maybe the restriction to "start" and "finish" graphs would make that acceptable - what do you think Oliver? I think the runahead limit implementation is OK. It's opt-in, we can explain the potential downside and that
Cool that our syntax support that though. But inter-cycle triggers would make that even worse (the offset would break pre-initial ignore). |
Long story short, I also think this is better:
If you look at the description of #4903 that's what I was aiming at, but maybe I shot myself in the foot there by trying to generalize it! |
Sorry!
Yuck - hadn't thought of that. |
I'm ok with separate recurrences for the ICP/FCP. I think a fully abstracted multi-graph solution would require a lot more thought than we can reasonably provide at this point in time. Long term it might also be the case that workflow modules / composition (e.g. the advanced handling of sub-workflows) may provide a simpler solution to this sort of problem.
It's definitely better, the issue is trying to work out how to implement it! The "start" and "finish" cycles need to be assigned a "cycle point" but what should that be? We could do something like subtract |
I was thinking along those lines, but I haven't thought through it in depth yet. |
Superseded by #5090 |
A bit of late night fun (sad!).
Implement "start-up and shutdown graphs" (effectively isolated from the main graph so that perpetual dependence on R1 & $ tasks can be avoided) ... by manipulating the runahead limit.
These changes close #4912
Imagine a task
prep
is supposed to prepare the run directory for all tasks; andclean
is supposed to tidy up at the end after all tasks have finished:With all tasks having the same run length, on master (without the new config items)
1/prep
runs at the same time as2/foo, 3/foo, 4/foo, 5/foo
, then1/foo
runs at the same time as5/clean
. Which is clearly not the intention, so we'd need additional ugly dependencies to make it work.Or ... on this PR branch,
1/prep => 1/foo
runs to completion, then2/foo, 3/foo, 4/foo
run to completion, then5/foo => 5/clean
runs. 💥Requirements check-list
CONTRIBUTING.md
and added my name as a Code Contributor.setup.cfg
andconda-environment.yml
.